Combining Planning and Dialog for Cooperative Assembly Construction

This article presents work on integrating a dialog component with a reactive planner into an assembly construction system. Using a re-active planner with a tight connection to the dialog component enables the system to ask for user assistance when encountering execution failures. We developed an action detection based on vision to monitor the execution of robotic and human actions. The combination of the robot-independent action detection with the reactive planner allows to correct many errors automatically (e.g. robot looses part and the detection process knows where the lost part is in the scene) and to incorporate human actions for cooperative plan execution (e.g. to resolve robotic failures). Additionally, monitoring actions allows to learn high-level assembly plans during their construction.